Service provision device, and method

ABSTRACT

A non-transitory recording medium storing a program that causes a computer to execute a process, the process includes: imaging a given object from plural different angles, and extracting from the plural obtained captured images, one or plural captured images having a feature amount that differs from a feature amount in another captured image by more than a specific reference amount; and providing the one or the plural extracted captured images as determination-use images employable in determination as to whether or not the given object is included in a captured image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2014-136745, filed on Jul. 2,2014, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a service provisiondevice, and method.

BACKGROUND

Recently, development is progressing in methods for determining whetheror not a specific subject is included in an image captured by camerasand the like. Such image determinations generally employ a method thatexecutes matching between a captured image and an image of the subjectserving as a reference for determination (determination-use image), anddetermines whether or not the subject is included in the captured image.

Commonly, the size and display angle of the subject included in thecaptured image is often different from the size and display angle in thedetermination-use image.

A method has therefore been proposed in which a portion of the capturedimage is set as a matching region, and determination is made as towhether or not a specific subject is included in a captured image byresizing and rotating the set matching region.

RELATED PATENT DOCUMENTS

Japanese Patent Application Laid-Open (JP-A) No. 2008-52598

SUMMARY

According to an aspect of the embodiments, a non-transitory recordingmedium stores a program that causes a computer to execute process. Theprocess includes: extracting, from out of plural captured imagesobtained by imaging a given object from plural different angles, one orplural captured images having a feature amount that differs by more thana specific reference amount from feature amounts of the other capturedimages; and providing the extracted one or plural captured images asdetermination-use image(s) employable in determination as to whether ornot the given object is included in a captured image.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a captured image.

FIG. 2 is a diagram for explaining differences in the appearance of anobject due to differences in imaging angle.

FIG. 3 is a diagram illustrating an example of a service provisiondevice.

FIG. 4 is a diagram illustrating an example of extraction processing ofa reference image.

FIG. 5 is a diagram illustrating an example of a database structure fora determination-use image.

FIG. 6 is a diagram illustrating an example of a database structure fora determination-use image.

FIG. 7 is a diagram illustrating an example of a service provisiondevice implemented by a computer.

FIG. 8 is a diagram illustrating a flow of determination-use imagegeneration processing.

FIG. 9 is flowchart illustrating an example of a flow ofdetermination-use image generation processing of a service provisionsystem according to a first exemplary embodiment.

FIG. 10 is a diagram illustrating an example of an execution result ofdetermination-use image generation processing of a service provisionsystem according to the first exemplary embodiment.

FIG. 11 is a flowchart illustrating an example of a flow ofdetermination processing of a service provision system according to thefirst exemplary embodiment.

FIG. 12 is a flowchart illustrating an example of a flow ofdetermination-use image generation processing of a service provisionsystem according to a second exemplary embodiment.

FIG. 13 is a diagram illustrating an example of an execution result ofdetermination-use image generation processing of a service provisionsystem according to the second exemplary embodiment.

FIG. 14 is a flowchart illustrating an example of a flow ofdetermination processing of a service provision system according to thesecond exemplary embodiment.

DESCRIPTION OF EMBODIMENTS

Detailed explanation follows regarding an example of an exemplaryembodiment according to technology disclosed herein with reference tothe drawings.

Recently, social networking services (SNS) that share informationbetween somewhat interlinked communities are being extensively used as acommunity instrument for individuals and corporations. In SNS,information is spread within a community using text, images, audio,etc., and this is becoming recognized as an important advertising mediumfor corporate enterprises. At the same time, opportunities areincreasing to implement campaigns using SNS and aimed at promoting salesof products.

In the present exemplary embodiment, an example of a system is explainedthat determines whether or not a product sold by a given manufacturer isincluded in an image when, for example, a user posts an image capturedof the product to the SNS, and then provides a gift or the like to theuser that posted the image taken of the product. Although the type ofproduct made the subject of a campaign is not limited, explanationfollows of an example in which a product A that is a canned drink soldby a drinks manufacturer serves as the product subject to a campaign.

In implementing such a campaign, determination needs to be made as towhether or not the product A is included in a captured image posted toan SNS by a user, such as that illustrated in FIG. 1. As is apparentfrom FIG. 1, the product A is sometimes captured from various angles,and there are many problems in determining whether or not the product Ais included in the captured image if the imaging angle is different,despite the product A being the same.

Although the product A is surrounded by a rectangle to make the positionof the product A in the captured image clear in the example of FIG. 1,there is no rectangle surrounding the product A in the captured imageposted to the SNS by the user.

FIG. 2 is a diagram using a sculpture to explain differences inappearance when the same object is imaged from different angles in aneasily understood manner. Image A is an image captured from the front ofthe sculpture, and image B is an image captured from the side of thesame sculpture. In this manner, when the imaging angle is different,despite this being the same sculpture, the hues change due to the lightilluminating the sculpture being brighter or duller, and the outlineshape, position, etc. of the sculpture also differ. Therefore featureamounts extracted from each image using known feature extractionalgorithms will also differ. It is therefore conceivable that theproduct A depicted in the captured image will be mis-determined as notbeing product A when the image capture conditions for product A differ.

A permissible range of difference in feature amounts for determinationof the product A is therefore set high, such that the product A isdetermined even if feature amounts of a determination-use imagepre-prepared to serve as a reference for determining whether or not anobject depicted in the captured image is the product A differ greatlyfrom feature amounts of the product A depicted in the captured image.However, when such setting is performed, a reverse case is nowconceivable in which a product different from the product A ismis-determined as the product A, making it difficult to call this aneffective method of reducing mis-determinations.

A method is conceivable in which plural determination-use imagescaptured at various imaging angles are pre-prepared, and the featureamounts of each of the determination-use images are compared to thefeature amounts of the product A depicted in a captured image to preventthe misrecognitions described above.

However, in this case calculation needs to be made of image similaritiesbetween a single captured image and each of plural determination-useimages, causing an issue in that the calculation volume increases as thenumber of determination-use images increases. It is therefore understoodthat performing determination of image similarity using as fewdetermination-use images as possible is preferable, though it remainsunclear how many determination-use images are preferably prepared, andfrom what angles they are preferably imaged.

Hence each of the following exemplary embodiments explain a system thatcreates determination-use images for determining whether a subject isincluded in a captured image by using a smaller number ofdetermination-use images.

First Exemplary Embodiment

FIG. 3 is a diagram illustrating an example of a service provisionsystem 10 according to an exemplary embodiment.

The service provision system 10 is a system in which a service provisiondevice 20, a user terminal 30, a social networking service server 40(referred to as SNS server 40 hereafter), and a service request terminal50 are connected to one another through a communication line 60. Notethat although the communication line 60 according to the presentexemplary embodiment is an internet line in the following explanation,the type of the communication line 60 is not limited thereto. Forexample, the communication line 60 may be a dedicated line, or may be anintranet such as a company-wide LAN. The architecture of thecommunication line 60 may be formed using any out of wires, wireless, ora mixture of wires and wireless.

The user terminal 30 is a terminal that posts captured images of theproduct A taken by the user to an SNS. The user terminal 30 is, forexample, a personal computer (PC), a notebook PC, a tablet terminal, ora smartphone, and may be any information device that includes a functionfor sending captured image data to the SNS server 40 through thecommunication line 60.

The SNS server 40 is a server that stores captured images sent from theuser terminal 30, and manages the publication destination and the likeof the captured image based on a security policy set by the user whosent the captured image.

The service request terminal 50 is, for example, installed at the drinksmanufacturer implementing the campaign for the product A, and exchangesinformation needed for the campaign for the product A with the serviceprovision device 20 described below. A browser pre-installed to theservice request terminal 50 may be employed for data exchange with theservice provision device 20, or a dedicated application provided by theadministrator managing the service provision device 20 may be employed.The installation location of the service request terminal 50 is notlimited to within the corporate entity implementing the campaign, and itgoes without saying that the service request terminal 50 may beinstalled at any location depending on the situation.

The user terminal 30, the SNS server 40, and the service requestterminal 50 may each have plural connections to the communication line60.

The service provision device 20 includes a communications section 21, aprovision section 22, a determination section 23, and a documentdatabase 24. The document database 24 is referred to as database 24hereafter.

The communications section 21 connects the service provision device 20to the communication line 60, and exchanges the needed data withinformation devices such as the user terminal 30, the SNS server 40, andthe service request terminal 50.

Specifically, the communications section 21 includes a reception section25 and an output section 26. The reception section 25 receives imagedata and the like from the user terminal 30, the SNS server 40, and theservice request terminal 50. The output section 26 outputs data includedin the service provision device 20 to the user terminal 30, the SNSserver 40, and the service request terminal 50 if necessary.

For example, the drinks manufacturer transmits images (registeredimages), captured while moving at least 360° around the product A, fromthe service request terminal 50 to the reception section 25 of theservice provision device 20 over the communication line 60 as thereferences for the campaign for the product A. The reception section 25receives the registered images capturing the product A from pluraldifferent angles. The registered images transmitted from the servicerequest terminal 50 to the service provision device 20 may be eitherstill images or video. The registered images in the present exemplaryembodiment are video as an example. The registered images include pluralcaptured images allocated frame numbers from FRAME 1 to FRAME M (where Mis a natural number) in capture time sequence, and the video isconfigured from a collection of the captured images corresponding toeach of the frames.

The provision section 22 includes an extraction section 27. Theextraction section 27 extracts captured images from the registeredimages received by the reception section 25 for use in determination asto whether or not the product A is included in the captured image postedto the SNS server 40 by the user.

First, specific explanation follows regarding extraction of the imagesby the extraction section 27. FIG. 4 is a diagram illustrating anexample of captured image extraction processing in the extractionsection 27.

First, the extraction section 27 sets as a reference image a capturedimage of a freely selected frame included in registered images receivedfrom the reception section 25. The extraction section 27 then designatescaptured images of frames that are different from the reference image ascandidate images for the reference images, and extracts feature amountsfrom the reference image and each of the candidate images using a knownmatching feature extraction algorithm.

Binary Robust Invariant Scalable Keypoints (BRISK) is employed as anexample of a feature extraction algorithm in the extraction section 27according to the present exemplary embodiment; however, the featureextraction algorithm employed by the extraction section 27 is notlimited thereto. For example, a feature extraction algorithm that excelsin extracting features from rotated images or the like may be employedaccording to the image capture conditions of the product that is thecampaign subject, such as conditions in which the product is oftencaptured in a tilted state and is often captured outdoors.

Note that BRISK is a feature extraction algorithm in which scaleinvariability and rotation invariability are introduced to Binary RobustIndependent Elementary Features (BRIEF), which are focused ondifferences in luminance between two separated points of an image. BRISKextracts feature amounts from images having modified scale and rotationwith higher precision than Scale Invariant Feature Transform (SIFT) orSpeeded Up Robust Feature (SURF) that are typical feature extractionalgorithms for extracting from images feature amounts that are invarianton scaling and rotation.

A feature amount a of a candidate image extracted using a featureextraction algorithm is expressed as a vector quantity ̂a=[a₀, . . . ,a_(N)], and a feature amount b of a reference image is expressed as avector quantity ̂b=[b₀, . . . , b_(N)]. Herein, N+1 (0≦N<∞) representsthe dimensionality of the feature amount, and the ̂ symbol denotes avector.

When employed, the extraction section 27 calculates distances betweenextracted feature amounts extracted from two respective images, anddetermines that the reference image and the candidate image are similarimages when the value of the calculated distance is a predeterminedthreshold value (similarity determination threshold value S1) or lower.In the extraction section 27 according to the present exemplaryembodiment, for example, the Euclid distance, as expressed by L2distance indicated in Equation (1), is employed as a distance of featureamounts representing image similarity.

$\begin{matrix}{{d\; {L_{2}\left( {{\hat{}a},{\hat{}b}} \right)}} = \sqrt{\sum\limits_{i = 0}^{N}\left( {a_{i} - b_{i}} \right)^{2}}} & (1)\end{matrix}$

Herein, dL₂(̂a, ̂b) denotes an L2 distance between the feature amount aand the feature amount b. Accordingly, the similarity between thereference image and the candidate image increases as the value of dL₂(̂a,̂b) decreases.

The determination of similarity between images is not limited to methodsemploying the L2 distance. For example, the similarity between thereference image and the candidate image may be determined by mappingfeature amounts extracted from each image at feature points, which areat locations to which feature extraction was applied, to vector space,and then comparing the vector distribution of the feature points of eachimage using statistical methods, machine learning, or the like.

Any indicator capable of quantitatively indicating the extent ofsimilarity between images, such as the distance L_(p) indicated inEquation (2), may be employed in the determination of extent ofsimilarity between images.

$\begin{matrix}{{d\; {L_{p}\left( {{\hat{}a},{\hat{}b}} \right)}} = \left( {\sum\limits_{i = 0}^{N}{{a_{i} - b_{i}}}^{p}} \right)^{\frac{1}{p}}} & (2)\end{matrix}$

Herein, dL_(p)(̂a, ̂b) denotes a distance L_(p) between the featureamount a and the feature amount b.

When the calculated L2 distance is the similarity determinationthreshold value S1 or below, namely, when the reference image and thecandidate image are similar, the extraction section 27 then selects fromthe registered images a frame image not yet selected as a candidateimage as a new candidate image. Then, the above described determinationof the extent of similarity is repeated between the reference images andthe new candidate image.

However, when the calculated L2 distance is greater than the similaritydetermination threshold value S1, namely, when the reference image andthe candidate image are dissimilar, the extraction section 27 sets thecandidate image as a new reference image. A frame image not yet selectedfrom the registered images as a candidate image is then selected as anew candidate image, and the above described determination of the extentof similarity between the new reference image and the new candidateimage is repeated.

The extraction section 27 thereby extracts from the registered imagesany reference images with image feature amounts that differ from eachother by more than the similarity determination threshold value S1.

The provision section 22 stores at least one reference image extractedby the extraction section 27 in the database 24 as a determination-useimage to be employed in the determination as to whether or not theproduct A is included in the captured image posted to the SNS server 40by the user, and provides the at least one determination-use image tothe determination section 23, described below.

The at least one or more determination-use image corresponding to theproduct A and provided by the provision section 22 is stored in thedatabase 24. The database 24 employs a document database, typicallyMongoDB for example. MongoDB is a document database that includes pluraldocuments in respective collections, and can define freely selectedfields for each document according to the structure and data format ofthe determination-use image. MongoDB is a database applied formanagement of so-called big data, in which database distribution as theamount of stored determination-use images increases is relatively simplecompared to in a conventional relational database (RDB).

FIG. 5 and FIG. 6 are diagrams illustrating an example of a databasestructure for different determination-use images in each diagram. Notethat any text following “//” in FIG. 5 and FIG. 6 is a commentexplaining the content of the corresponding field.

The database 24 according to the present exemplary embodiment includes,for example, a primary key, a time of storage, photographer information,information regarding the imaged object, information regarding thefeature extraction algorithm employed, the feature amount in the image,the determination algorithm of the reference image, informationregarding effects applied to the image, image storage location, and thelike.

In the database structure illustrated in FIG. 5 and FIG. 6, thedifferences between FIG. 5 and FIG. 6 are the field portions surroundedby dashed lines in FIG. 6. In this case, the primary key, the time ofstorage, the feature amount of the image, and the storage location aredifferent in the database.

Although the database 24 according to the present exemplary embodimentemploys MongoDB in order to store determination-use images, the employeddatabase is not limited thereto, and other types of databases may beemployed. Moreover, a known file system such as the NT File System(NTFS) may be employed without employing a database.

The reception section 25 receives captured images posted to the SNSserver 40 during the campaign period for the product A.

The determination section 23 receives captured images from the receptionsection 25 and extracts feature amounts from the captured images usingthe same feature extraction algorithm as the extraction section 27. Thedetermination section 23 then, for example, calculates an L2 distanceindicating the extent of similarity of the image from the featureamounts of the captured image and the feature amounts of each of thedetermination-use images of the product A stored in the database 24using Equation (1). The determination section 23 then determines whetheror not the product A is included in the captured image by comparing thecalculated L2 distance and the similarity determination threshold valueS1.

According to the determination result made by the determination section23, the output section 26 outputs an email or the like containinginformation related to the campaign for the product A, such as a URLlink to an application form for a gift, to the user terminal 30 of theuser who posted the captured image including the product A for example.In such an event, the output section 26 may output informationspecifying the determination result made by the determination section 23and the provision origin of the captured image, to the service requestterminal 50 of the drinks manufacturer implementing the campaign for theproduct A. The reception section 25 may acquire the captured imagemanaged by the SNS server 40, and information such as an email addressspecifying the provider of the captured image, using applicationprogramming interfaces (API) pre-prepared for each SNS service.

FIG. 7 illustrates a computer system 100 as a computer implementableexample of the service provision device 20, the user terminal 30, theSNS server 40, and the service request terminal 50 included in theservice provision system 10.

The computer system 100 illustrated in FIG. 7 as the service provisionsystem 10 includes a computer 200 as the service provision device 20.The computer system 100 also includes a computer 300 as the userterminal 30, a computer 400 as the SNS server 40, and a computer 500 asthe service request terminal 50.

The computer 200 includes a CPU 202, memory 204, and a nonvolatilestorage section 206. The CPU 202, the memory 204, and the nonvolatilestorage section 206 are connected to one another through a bus 208. Thecomputer 200 includes an input section 210 such as a keyboard and mouse,and a display section 212 such as a display. The input section 210 andthe display section 212 are connected to the bus 208. An IO 214 forreading/writing from/to a recording medium 232 is also connected to thebus 208 in the computer 200. The computer 200 also includes acommunications interface (IF) as an interface for connecting to thecommunication line 60, and the communications IF 216 is also connectedto the bus 208. The storage section 206 may be implemented by a harddisk drive (HDD), flash memory, or the like.

A service provision program 218 that causes the computer 200 to functionas the service provision device 20 illustrated in FIG. 3, a similaritydetermination information storage region 228, and a database storageregion 230 are stored in the storage section 206. The service provisionprogram 218 stored in the storage section 206 includes a communicationsprocess 220, a provision process 222, and a determination process 224.

The CPU 202 reads the service provision program 218 from the storagesection 206, expands the service provision program 218 into the memory204, and executes each process included in the service provision program218. The CPU 202 expands similarity determination information includedin the similarity determination information storage region 228 into thememory 204 as the similarity determination threshold value S1. The CPU202 expands information for configuring a database included in thedatabase storage region 230 into the memory 204, and configures thedatabase 24.

The computer 200 operates as the service provision device 20 illustratedin FIG. 3 by the CPU 202 reading the service provision program 218 fromthe storage section 206, expanding the service provision program 218into the memory 204, and executing the service provision program 218.The computer 200 operates as the communications section 21 including thereception section 25 and the output section 26 illustrated in FIG. 3 bythe CPU 202 executing the communications process 220. The computer 200operates as the provision section 22 included in the extraction section27 illustrated in FIG. 3 by the CPU 202 executing the provision process222. The computer 200 operates as the determination section 23illustrated in FIG. 3 by the CPU 202 executing the determination process224.

Note that the service provision device 20 may also be implemented by,for example, a semiconductor integrated circuit, and more specificallyby an application specific integrated circuit (ASIC), or the like.

Next, explanation follows regarding operation of the service provisiondevice 20 according to the present exemplary embodiment. Whendetermination-use images of the product A are not stored in the database24, the service provision device 20 according to the present exemplaryembodiment executes determination-use image generation processing.

FIG. 8 is a diagram schematically illustrating an example of a flow ofdetermination-use image generation processing. For example, asillustrated in FIG. 8, the reception section 25 receives registeredimages that were captured while a representative of the manufacturermoved around a product from a position X→a position Y→a position Z→theposition X in this order, by at least 360° or more. The extractionsection 27 then extracts at least one or more images out of theregistered images that are dissimilar to one another; these arereferences image A to reference image C in the example of FIG. 8. Theprovision section 22 then stores the reference images extracted from theextraction section 27 in the database 24 as determination-use images.

FIG. 9 is a flowchart illustrating an example of a flow of thedetermination-use image generation processing schematically explained inFIG. 8.

First, at step S10, initialization processing needed for execution ofthe determination-use image generation processing is executed.Specifically, the extraction section 27 acquires the similaritydetermination threshold value S1 expanded in to the memory 204.

At step S20, the reception section 25 determines whether or not theregistered images of the product A were received from service requestterminal 50. In cases of negative determination, the processing of stepS20 is repeated until the registered images are received. In cases ofaffirmative determination, the received registered images are stored ina predetermined region in the memory 204, and processing transitions tostep S30.

At step S30, the extraction section 27 extracts the first frame includedin the registered images received by the processing of step S20, namely,the captured image corresponding to FRAME 1, and sets the extractedframe as a reference image. At this time the extraction section 27stores the frame number of the captured image extracted from theregistered images in a predetermined region in the memory 204.

At step S40, the extraction section 27 updates the frame number of thecaptured image next to be extracted from the registered images to thevalue given by adding 1 to the frame number stored in the memory 204.

At step S50, the extraction section 27 extracts the captured image ofthe frame number updated in the processing of step S40 from theregistered images, and determines whether or not extraction of thecaptured image succeeded. Processing then transitions to step S60 incases of affirmative determination. Note that image extraction failswhen an attempt is made to extract from the registered images a capturedimage of a frame number exceeding the final frame number included in theregistered images.

At step S60, the extraction section 27 uses the BRISK method to extractthe captured images extracted from the registered images in theprocessing of step S50, namely, the extraction section 27 extracts fromeach of the images the feature amounts ̂a of the candidate images, andthe feature amounts ̂b of the currently set reference images. Asmentioned above, the feature extraction algorithm employed by theextraction section 27 is not limited to the BRISK method, and knownfeature extraction algorithms may be employed.

Then, according to Equation (1), the extraction section 27 calculatesthe L2 distance dL₂ (̂a, ̂b) indicating the extent of similarity of theimages from the feature amounts ̂a of the candidate images and thefeature amounts ̂b of the reference images. As mentioned above, theindicator of the extent of similarity between images employed by theextraction section 27 is not limited to the L2 distance, and anotherindicator capable of quantitatively indicating the extent of similaritybetween images may be employed.

Effect processing on images, such as background elimination, mayexecuted on the candidate images and the reference images beforeextracting the feature amounts from the candidate images and thereference images. In such cases the feature amounts of the image of theproduct A can be extracted with high precision since the backgroundbehind the product A is eliminated from each of the images.

At step S70, the extraction section 27 determines whether or not the L2distance dL₂ (̂a, ̂b) calculated at step S60 is greater than thesimilarity determination threshold value S1 acquired in the processingof step S10, namely, the extraction section 27 determines whether or notthe candidate image and the reference image are similar to each other.When the reference image and the candidate image are determined to besimilar to each other, this means that, in other words, it is possibleto determine that a reference image and the candidate image capturedfrom a different angle are similar images to each other using anexisting reference image. Accordingly, in cases of negativedetermination, namely, cases of the reference image and the candidateimage being similar to each other, processing transitions to step S40without the candidate image subject to determination being set as areference image.

Processing transitions to step S80 when affirmative determination ismade by the determination processing of step S70. When the candidateimage and the reference image are determined as not being similar toeach other, this means that, in other words, it is not possible toextract the features of the product A from the candidate image using theexisting reference images even though the candidate image depicts thesame product A. This is because the reference images and the candidateimage were captured under different image capture conditions, such asdifferent imaging angles, and the feature amounts extracted from theimages therefore differ by more than the similarity determinationthreshold value S1 even though they are images of the same product A.

Thus at step S80, instead of just the existing reference images, theextraction section 27 sets the candidate image determined as not beingsimilar to the existing reference images as a new reference image, andprocessing transitions to step S40. The processing of step S40 to stepS80 is then repeated, and any reference images of the product A areacquired having differences in the feature amounts of the captured imagediffering from captured images of the product A captured in variousdirections along the periphery of the product A by more than thesimilarity determination threshold value S1.

When negative determination is made by the determination processing ofstep S50, namely, when all of the reference images have betweenextracted from the registered images, processing transitions to stepS90.

At step S90, the provision section 22 stores as determination-use imagesall of the reference images set by the extraction section 27 in theprocessing of step S30 and step S80, in a predetermined region in thememory 204. Since the image of FRAME 1 is set as a reference image inthe processing of step S30, at least one or more determination-use imageexists.

The determination-use image generation processing thus ends.

Determination is made as to whether or not extraction of all of thereference images from the registered images is complete according towhether or not the extraction of captured images succeeded in theprocessing of step S50. However, this determination may be made by othermethods. For example, the final frame number for the captured imagesincluded in the pre-registered images may be acquired, and determinationmay be made as to whether or not the frame number updated by theprocessing of step S40 is the final frame number or less.

In the determination-use image generation processing illustrated in FIG.9, at the processing of step S80 the reference images for comparisonwith the candidate image are successively updated when the differencethat is the distance between the feature amounts of the candidate imageand feature amounts of the reference image are greater than thesimilarity determination threshold value S1. However, thedetermination-use images may be generated by each time comparing againstthe reference images set by the processing of step S30, without updatingthe reference images being compared to the candidate images.

In such cases, in the processing of step S80, the extraction section 27,for example, stores candidate images determined to not be similar to theexisting reference images in the memory 204, and changes the value ofthe similarity determination threshold value S1 employed in thedetermination processing of step S70 each time the processing of stepS80 is executed. For example, if the value of similarity determinationthreshold value S1 is provisionally set as K for generatingdetermination-use images with image feature amounts that different fromone another by more than similarity determination threshold value S1,each time the processing of step S80 is executed, the value of thesimilarity determination threshold value S1 may be changed in thesequence K, (K×2), (K×3), and so on.

The provision section 22 may then, in the processing of step S90, setthe reference images set by the processing of step S30 and the candidateimages saved in the memory 204 by the processing of step S80 asdetermination-use images.

FIG. 10 is a diagram illustrating an example of a determination-useimage extracted from the registered images as a result of thedetermination-use image generation processing illustrated in FIG. 9. Inthe example of FIG. 10, four images, determination-use image A todetermination-use image D, are extracted from the registered images. Thedetermination-use image A to determination-use image D extracted by thedetermination-use image generation processing are images having imagefeature amounts that differ from one another by more than the similaritydetermination threshold value S1.

FIG. 11 is a flowchart illustrating an example of a flow ofdetermination processing executed by the service provision device 20during the campaign period for the product A after the determination-useimage generation processing illustrated in FIG. 9 ends.

First, at step S100, the reception section 25 references the SNS server40 and determines whether or not there are captured images posted to theSNS server 40 by a user. The processing of step S100 is repeated and theSNS server 40 is continuously referenced in cases of negativedetermination. In cases of affirmative determination, the capturedimages are acquired from the SNS server 40 and stored in a predeterminedregion in the memory 204, and processing transitions to step S120. Whendoing so, the reception section 25 uses an API, provided by theadministrator or the like managing the SNS server 40, to acquireidentification information uniquely indicating the user who posted theacquired captured image, for example an email address, and stores theidentification information in the memory 204 in association with thecaptured image.

At step S120, the determination section 23 determines whether or notthere are any determination-use images not yet acquired from thedetermination-use images of the product A stored in the database 24, andprocessing transitions to step S140 when affirmative determination ismade.

At step S140, the determination section 23 acquires determination-useimages not yet acquired from the database 24 along with feature amountsof the determination-use images.

Then, at step S150, the determination section 23 performs image matchingby scanning the captured images acquired by the processing of step S100while resizing the determination-use images acquired by the processingof step S140. The determination section 23 then uses the same featureextraction algorithm as the feature extraction algorithm used by theextraction section 27 to calculate the feature amounts of the regions ofthe captured images determined to be most similar to thedetermination-use images.

The determination section 23 then calculates, for example according toEquation (1), the L2 distance indicating the extent of mutual imagesimilarity from the feature amounts of the captured image extracted inthe current step and from the feature amounts of the determination-useimages acquired by the processing of step S140. Although the L2distances between the captured images and the determination-use imagesare calculated here, an indicator of the extent of image similarity maybe calculated other than the L2 distance.

At step S160, the determination section 23 acquires the similaritydetermination threshold value S1 expanded in the memory 204, anddetermines whether or not the L2 distance calculated at step S150 is thesimilarity determination threshold value S1 or less. In cases ofaffirmative determination, namely, cases in which determination is madethat there are no regions in the captured image similar to thedetermination-use images of the product A, processing transitions tostep S120, the processing of step S120 to step S160 is repeated, anddetermination is made as to whether there is a region in the capturedimage similar to the determination-use image of the product A.Processing transitions to step S170 when the determination result of thecurrent step is an affirmative determination, namely, when determinationis made that there is a region in the captured image similar to adetermination-use image for the product A.

At step S170, the determination section 23 determines that the product Ais included in the captured image since a portion of the captured imageand a determination-use image for the product A are similar to eachother.

Then, at step S180, the determination section 23 requests that theoutput section 26 output an email containing information relating to thecampaign for the product A such as a URL link to an application form fora gift to the user who uploaded the captured image. The output section26 generates the email based on the request from the determinationsection 23 and outputs the generated email to the email address that wasacquired by the processing of step S100 of the user who posted thecaptured image, and the determination processing illustrated in FIG. 11ends.

When negative determination is made in the determination processing ofstep S120, namely, when determination is made that there are no regionsin the captured image similar to any of the determination-use images forthe product A, processing transitions to step S130. Then, at step S130,the determination section 23 determines that the product A is notincluded in the captured image, and the determination processingillustrated in FIG. 11 ends.

The determination processing illustrated in FIG. 11 is repeatedlyexecuted during the campaign period for the product A.

As a result of the determination processing, the user of the userterminal 30 that received the email from the service provision device 20may, for example, obtain a reward such as a gift from the drinksmanufacturer by accessing the URL in the email and inputting themandatory information into the application form.

Although an email is sent to the user who posted the captured image eachtime determination is made that the product A is included in a capturedimage in the determination processing illustrated in FIG. 11, the timingat which the email is sent is not limited thereto. For example, theemail may be saved in the memory 204, and the email may be sent to theuser who posted the captured image after the campaign period for theproduct A ends.

In this manner, the service provision device 20 may employ images withfeature amounts that differ from one another by more than the similaritydetermination threshold value S1 as determination-use images whendetermining whether or not the product A is included in the capturedimage. Accordingly, the product A may be extracted with high precisionfrom the captured images under different image capture conditions usinga smaller number of determination-use images.

Explanation has been given above in which the registered images receivedfrom the service request terminal 50 are images captured while movingaround the product A by at least 360° or more; however, the registeredimages may be images capturing the product A from various differentangles. In such cases, determination-use images may be generated inorder to extract the product A included in captured images with higherprecision.

Only pre-captured images of the product A for which image capture hascompleted are employed as registered images in the service provisiondevice 20 according to the present exemplary embodiment; however, theregistered images are not limited to such images. For example, theregistered images may be real-time images captured while rotating oncearound the product A, sent from a terminal (an image capture terminal)installed with a dedicated application provided by the administrator whomanages the service provision device 20. The service request terminal 50may be employed as the image capture terminal.

In such cases, the reception section 25 receives instructions from theimage capture terminal indicating image capture start or image captureend, and starts acquiring registered images on instruction to startimage capture. Then, at step S40 in FIG. 9, the extraction section 27acquires images one frame at a time, and in the processing of step S50,the extraction section 27 determines whether or not instruction to endimage capture has been received from the image capture terminal, andprocessing transitions to step S90 and the determination-use imagegeneration processing is ended if instruction to end image capture hasbeen received.

Moreover, when a photographer of images of the product A moves aroundthe product A, determination-use image generation processing may beended at the timing when the photographer returns to the image capturestart location, without receiving an instruction to end image capturefrom the image capture terminal. In such cases, it may, for example, bedetermined that imaging has returned to the image capture start locationwhen the feature amounts of the image received from the image captureterminal in the processing of step 40 is compared to the feature amountsof each of the reference images set in the processing of step 30, anddiffers from by a predetermined value or less.

Second Exemplary Embodiment

Next, explanation follows regarding a second exemplary embodiment. Theservice provision system according to the second exemplary embodiment issimilar to the service provision system according to the first exemplaryembodiment illustrated in FIG. 3. The service provision system accordingto the second exemplary embodiment may therefore be implemented by acomputer system similar to the computer system 100 according to thefirst exemplary embodiment illustrated in FIG. 7. However, the serviceprovision device 20 in FIG. 3 is replaced with a service provisiondevice 20A, the extraction section 27 is replaced with an extractionsection 27A, and the determination section 23 is replaced with adetermination section 23A. Moreover, in FIG. 7, the computer 200 isreplaced with a computer 200A, the provision process 222 is replacedwith a provision process 222A, the determination process 224 is replacedwith a determination process 224A, and the service provision program 218is replaced with a service provision program 218A.

Herein, the same reference numerals are allocated to portionscorresponding to those of the first exemplary embodiment, andexplanation focuses on the portions that differ from the first exemplaryembodiment.

In the extraction section 27 according to the first exemplaryembodiment, a known feature extraction algorithm is employed to extractfeature amounts from each of the reference images and the candidateimage, and determination is made as to whether or not the candidateimage is to be set as a new reference image, based on distances betweenthe feature amounts.

In the extraction section 27A according to the present exemplaryembodiment, the reference images are set by categorizing the candidateimages using a clustering algorithm. Clustering is a type of imageprocessing that categorizes images using a quality indicative of theexistence of similar images clustering together spatially in featurespace.

The clustering algorithm employed by the extraction section 27A is notlimited, and for example, a known clustering algorithm such as a nearestneighbor method may be employed.

In a nearest neighbor method, an arbitrary number of classes arepre-generated, and each class is allocated a typical vector value, suchas a feature amount, representing the class. Each candidate image isthen categorized into a class having the typical vector value thesmallest distance away from the vector value, such as the featureamount, obtained from the candidate image, and wherein this distance isthe similarity determination threshold value S1 or lower. When thedistance from feature amount of the candidate image to the typicalvector of the closest class is greater than similarity determinationthreshold value S1, the feature amount of the candidate image is set asa typical vector value, and a new class is generated. Images are thenselected one at a time from each class and set as registered images.

The determination section 23A according to the second exemplaryembodiment also uses the same clustering algorithm as the extractionsection 27A to categorize the captured image into a class ofdetermination-use images having feature amounts that are typical vectorvalues. When doing so, the determination section 23A determines that theproduct A is included in the captured image when it is able tocategorize the captured image into a class, and determines that theproduct A is not included in the captured image when unable tocategorize the product A into any class.

Next, explanation follows regarding operation of the service provisiondevice 20A according to the present exemplary embodiment. The serviceprovision device 20A according to the present exemplary embodimentexecutes the determination-use image generation processing whendetermination-use images for the product A are not stored in thedatabase 24.

FIG. 12 is a flowchart illustrating an example of a flow ofdetermination-use image generation processing according to the presentexemplary embodiment. The flowchart illustrated in FIG. 12 differs fromthe flowchart of the determination-use image generation processingaccording to the first exemplary embodiment illustrated in FIG. 9 inthat initialization processing of step S10 is substituted by theinitialization processing of step S15, and that the processing of stepS30 is omitted. Moreover, the processing of step S60 is substituted bythe processing of step S55, and the processing of step S70 issubstituted by the processing of step S65.

First, at step S15, the extraction section 27A acquires the similaritydetermination threshold value S1 that was expanded into the memory 204,and acquires the typical vector values of plural arbitrary classes thatwere expanded into the memory 204 in advance. Typical vector informationfor each of the classes is stored in the storage section 206, and priorto starting the determination-use image generation processing, the CPU202 expands the typical vector information of each of the classes intothe memory 204 as the typical vector values. The typical vector valuesof each of the classes differ greatly from one another by, for example,three times the similarity determination threshold value S1.

Then, at step S55, the extraction section 27A extracts the featureamounts of a candidate image, and categorizes the candidate image using,for example, a known clustering algorithm such as the nearest neighbormethod described above.

As explained specially above, each candidate image is then categorizedinto a class such that the feature amount of the candidate image and thetypical vector value acquired in the processing of step S15 are thesmallest distance away, and wherein this distance is less than thesimilarity determination threshold value S1. When the distance betweenthe feature amount of the candidate image and the typical vector of theclosest class is greater than similarity determination threshold valueS1, the feature amount of the candidate image is set as a typical vectorvalue, and a new class is generated.

Effect processing on images, such as background elimination, may beexecuted on the candidate images before extracting the feature amountsfrom the candidate image. In such cases, the feature amounts of theimage of the product A can be extracted from the candidate image withhigh precision since the background behind the product A is eliminated.

At step S65, the extraction section 27A determines whether or not thecandidate image categorized into a class by the processing of step S55is the only candidate image in the categorization destination class.When affirmative determination is made, processing then transitions tostep S80, and the only image in that categorization destination class isset as a reference image.

When negative determination is made, processing transitions to step S40,and processing to categorize the candidate image of the updated framenumber into one of the classes is repeated.

In this manner, images of the registered images with image differencesbigger than the similarity determination threshold value S1 can beextracted as determination-use images also in cases in which theextraction section 27A uses the clustering algorithm.

FIG. 13 is a diagram illustrating an example of determination-use imagesextracted from the registered images as a result of thedetermination-use image generation processing illustrated in FIG. 12. Inthe example of FIG. 13, one determination-use image is extracted fromthe registered images for each of four classes, class A to class D. Thedetermination-use images for each class extracted by thedetermination-use image generation processing are images with mutualimage differences greater than the similarity determination thresholdvalue S1.

FIG. 14 is a flowchart illustrating an example of a flow ofdetermination processing executed during the campaign period for theproduct A by the service provision device 20A after thedetermination-use image generation processing illustrated in FIG. 12 hasended.

The flowchart illustrated in FIG. 14 differs from the flowchart of thedetermination processing according to the first exemplary embodimentillustrated in FIG. 11 in that the processing of step S110 has beenadded. Moreover, the processing of each of step S120, step S140, stepS150, and step S160 of FIG. 11 has been substituted by processing ofstep S115 and step S125.

At step S110, the determination section 23A calculates the featureamounts of the captured image acquired by the processing of step S100using the same feature extraction algorithm as was used by theextraction section 27A.

At step S115, the determination section 23A categorizes the capturedimage into the respective classes represented by the determination-useimages, based on the feature amounts of the captured image acquired bythe processing of step S110. When this is performed, the determinationsection 23A categorizes the captured images into the respective classesusing, for example, the nearest neighbor method used by the extractionsection 27A to extract the reference images from the registered images.The clustering algorithm employed by the determination section 23A isnot limited to the nearest neighbor method, and known clusteringalgorithms may be employed.

At step S125, the determination section 23A determines whether or not ithas been able to categorize the captured image into a class. Processingtransitions to step S130 when negative determination is made, andprocessing transitions to step S170 when affirmative determination ismade.

In this manner, the service provision device 20A is able to determinewhether or not the product A is included in a captured image usingimages having mutual image differences greater than the similaritydetermination threshold value S1 as the determination-use images. Theproduct A can thereby be extracted with high precision from capturedimages having different image capture conditions using a smaller numberof determination-use images.

Similarly to the service provision device 20 according to the firstexemplary embodiment, the service provision device 20A according to thepresent exemplary embodiment may also employ real-time images sent froman image capture terminal during image capture of the product A, insteadof using pre-captured images of the product A for which image capture isalready complete as the registered images.

Although explanation of technology disclosed herein has been given aboveusing exemplary embodiments, technology disclosed herein is not limitedto the scope of the exemplary embodiments above. Various modificationsand improvements may be made to the exemplary embodiments above within arange not departing from the spirit of technology disclosed herein, andthe technological scope of technology disclosed herein also encompassesmodes in which such modifications and improvements have been made. Forexample, the processing sequence may be modified within a range notdeparting from the spirit of technology disclosed herein.

Although explanation has been given above in which the service provisionprograms 218, 218A are pre-stored (pre-installed) on the storage section206, there is not limitation thereto. The service provision programsaccording to technology disclosed herein may also be provided in aformat recorded on a computer readable recording medium. For example,the service provision programs according to technology disclosed hereinmay also be provided in a format recorded on a portable recording mediumsuch as a CD-ROM, a DVD-ROM, or USB memory. The service provisionprograms according to technology disclosed herein may also be providedin a format recorded on, for example, semiconductor memory such as flashmemory.

In the first exemplary embodiment and the second exemplary embodiment,explanation has been given regarding an example in which the serviceprovision devices 20, 20A are applied to a sales promotion campaign fora product using the SNS, and determination is made as to whether or notthe product that is the campaign subject is included in images posted tothe SNS.

However, the areas of application of the service provision devices 20,20A are not limited to this example.

For example, the service provision devices 20, 20A may be employed in amarketing service utilizing the images posted to the SNS.

To simplify the explanation below, explanation is given of an examplesituation in which the service provision device 20 employs a marketingservice. However, the service provision device 20A may also be appliedin a similar situation.

First, the service provision device 20 executes the determination-useimage generation processing illustrated in FIG. 9, and generates adetermination-use image for the product (product subject to analysis)that is the subject of the marketing service. Note that there may beplural products subject to analysis, and determination-use images aregenerated for each of the products subject to analysis in such cases.

The service provision device 20 then acquires the captured image postedto the SNS server 40, and determines whether or not the product subjectto analysis is included in captured image by executing the determinationprocessing illustrated in FIG. 11.

When the product subject to analysis is included into the capturedimage, information related to when the product subject to analysis wasimaged is also acquired in addition to the captured image. The contentof the related information acquired is not limited, and includesinformation acquirable from the captured image.

For example, the related information may include the number of productssubject to analysis included in the captured image, the ratio of thearea occupied by products subject to analysis with respect to the sizeof the captured image, the image capture environment indicating whetherthe image capture location was indoors or outdoors, the weather at thetime of image capture, and the like. Moreover, the type of processingfor treating the captured image may be acquired, such as sepia toningsuch that an image exhibits the impression of passage of time, orstyling as an illustration. Moreover, information may be acquired thatindicates the number of people depicted with the product subject toanalysis, their estimated ages, and whether they are smiling or angry,and information related to fashion that indicates whether they areformally dressed or casually dressed.

The relation information may also include information related to theimage matching executed by the processing of step S150 illustrated inFIG. 11. The information related to the image matching is, for example,information indicating which location of the captured image was resizedand by what extent, and to which determination-use image it wasdetermined to be similar.

Then, based on the acquired related information related to the productsubject to analysis, the service provision device 20, for example,performs analysis of information indicating who tends to use the productsubject to analysis, and when and where they tend to use it, with thisinformation being valuable in sales promotion of the product subject toanalysis and the like. The analysis results are then sent to the servicerequest terminal 50 of the manufacturer that requested the markingservice.

When this is performed, the service provision device 20 may send therelated information to the service request terminal 50 along with theanalysis results. Moreover, the related information acquired from thecaptured images may be sent to the service request terminal 50 alone,and analysis based on the related information may implemented in theservice request terminal 50.

When the product subject to analysis is analyzed based on the relatedinformation, the service provision device 20 may analyze a combinationof other information such as text posted to the SNS server 40 togetherwith the captured image, and marking data related to the product subjectto analysis provided by the manufacturer.

In this manner, the service provision device 20 is able to provide themanufacturer with information related to sales promotion of the product.

Explanation follows regarding an example situation in which the serviceprovision devices 20, 20A are employed in an active support serviceemploying the images posted to the SNS. The active support serviceincludes, for example, a service that analyzes images that include theproduct, and forwards images that give a favorable reaction to manypeople, namely, images likely to prove popular, and thereby increasesfavorable feelings toward the manufacturer by users.

The active support service determines whether or not the product subjectto active support (the product subject to support) is included in thecaptured image by a method similar to that of the marking servicedescribed above.

When the product subject to support is included in the captured image,analysis is made as to whether or not the captured image is likely to bea popular image. In this analysis, for example, analysis is performedusing plural evaluation items such as the feelings of people depicted inthe captured image, the presence of absence of animals, and the contentof any modification processing performed on the captured image, and setsa score for each evaluation item. For example, captured images in whichthe depicted people are laughing are set with a higher score than thosein which the people are angry. The scores for each evaluation item arethen summed, and captured images with a predetermined score or greaterare determined to be images likely to be popular.

Images likely to be popular depicting the product of the manufacturerthat requested the active support service can be spread across theinternet, enabling the service provision device 20 to provide themanufacture with a service for increasing the favorable feelings towardthe manufacturer by users.

Explanation has been given in the exemplary embodiments of cases inwhich the service provision devices 20, 20A are implemented by singlecomputers 200, 200A. However, the processes may be executed by differentcomputers, and the service provision devices 20, 20A may be implementedby respective computers connected by the communication line 60, in adistributed processing configuration.

In such cases, the provision processes 222, 222A provide thedetermination-use images over the communication line 60 to the computersthat execute the determination processes 224, 224A.

Although the captured images are acquired from the SNS server 40 in eachexemplary embodiment, the acquisition source of the captured images isnot limited to the SNS server 40, and may be a server where users of anunspecified large number of user terminals 30 connected to thecommunication line 60 publish captured images, such as a message boardor home page.

Although each of the exemplary embodiments adopt modes in which theregistered images are received from the manufacturer, a request may bereceived from the manufacturer, and the registered images may becaptured by the service provider that manages the service provisiondevices 20, 20A.

The appearance of a subject included in captured images changesaccording to image capture conditions during image capture of thesubject, such as the angle and distance, even though the captured imagesare of the same subject, and there is a possibility of mis-recognizingthe subject as not being included in a captured image.

Thus, conventionally, a method is employed in which mis-recognition isprevented by varying the imaging angle and pre-preparing plural captureddetermination-use images, and comparing the feature amounts of each ofthe determination-use images with the feature amounts of the product Adepicted in the captured image. However, it is unclear how manydetermination-use images to prepare, and from what angles they are to becaptured for this purpose.

One aspect of technology disclosed herein exhibits an advantage effectof enabling an optimization to be achieved in data volume ofdetermination-use images for determining that a subject is included in acaptured image.

All examples and conditional language provided herein are intended forthe pedagogical purposes of aiding the reader in understanding theinvention and the concepts contributed by the inventor to further theart, and are not to be construed as limitations to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although one or more embodiments of thepresent invention have been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

What is claimed is:
 1. A non-transitory recording medium storing aprogram that causes a computer to execute a process, the processcomprising: extracting, from a plurality of captured images obtained byimaging a given object from a plurality of different angles, one or morecaptured images having a feature amount that differs by more than aspecific reference amount from feature amounts of the other capturedimages; and providing the extracted one or more captured images as oneor more determination-use images employable to determine whether or notthe given object is included in a captured image.
 2. A non-transitoryrecording medium storing a program that causes a computer to execute aprocess, the process comprising: from a plurality of captured imagesobtained by imaging a given object from a plurality of different angles,extracting and storing, in a memory, one or more captured images havinga feature amount that differs by more than a specific reference amountfrom feature amounts of the other captured images; and comparing afeature amount of a captured image other than the plurality of capturedimages against the feature amount of the one or more captured imagesstored in the memory; and determining whether or not the given object isincluded in the captured image other than the plurality of capturedimages based on a comparison result.
 3. The non-transitory recordingmedium of claim 2, wherein the captured image other than the pluralityof captured images is an image acquired through the Internet.
 4. Thenon-transitory recording medium of claim 2, wherein the process furthercomprises: identifying a provision source of the captured image otherthan the plurality of captured images in which the given object has beendetermined to be included; and outputting a message to the identifiedprovision source.
 5. The non-transitory recording medium of claim 4,wherein the process further comprises: outputting information thatidentifies the provision source of the captured image other than theplurality of captured images in which the given object has beendetermined to be included as a result of the determination.
 6. Thenon-transitory recording medium of claim 1, wherein the process furthercomprises: from among the plurality of captured images obtained byimaging from a plurality of different angles, setting a referencecaptured image as a reference, and extracting a captured image in whicha difference from a feature amount of the reference captured image ismore than the specific reference amount; and setting the extractedcaptured image as a new reference captured image, and further extractinga captured image in which a difference from a feature amount of the newreference captured image is more than the specific reference amount. 7.The non-transitory recording medium of claim 1, wherein the processfurther comprises: categorizing the plurality of captured imagesobtained by imaging from a plurality of different angles into aplurality of classes such that typical vectors of each of the classesdiffer by more than the specific reference amount from each other; andextracting one captured image from each class in the plurality ofclasses.
 8. The non-transitory recording medium of claim 1, wherein theplurality of captured images obtained by imaging from a plurality ofdifferent angles are each captured images in which a background behindthe given object has been eliminated.
 9. A service provision method,comprising: by a processor, extracting, from a plurality of capturedimages obtained by imaging a given object from a plurality of differentangles, one or more captured images having a feature amount that differsby more than a specific reference amount from feature amounts of theother captured images; and by the processor, providing the extracted oneor more captured images as one or more determination-use imagesemployable to determine whether or not the given object is included in acaptured image.
 10. The service provision method of claim 9, wherein: bythe processor, a reference captured image is set as a reference fromamong the plurality of captured images obtained by imaging from aplurality of different angles, and a captured image is extracted inwhich a difference from a feature amount of the reference captured imageis more than the specific reference amount; and by the processor, theextracted captured image is set as a new reference captured image, and acaptured image in which a difference from a feature amount of the newreference captured image is more than the specific reference amount isfurther extracted.
 11. The service provision method of claim 9, wherein:by the processor, the plurality of captured images obtained by imagingfrom a plurality of different angles are categorized into a plurality ofclasses such that typical vectors of each of the classes differ by morethan the specific reference amount from each other; and by theprocessor, one captured image is extracted from each class in theplurality of classes.
 12. The service provision method of claim 9,wherein the plurality of captured images obtained by imaging from aplurality of different angles are each captured images in which abackground behind the given object has been eliminated.